Integration Devleopment

RESTful APIs – An accurate description

Hi everyone

Today I thought I would do a quick post to cover RESTful APIs, and what they are. The reason for this article is that I have, on numerous occasions, encountered developers (and indeed whole teams) who have misunderstood this concept at its very core. This causes a number of problems, firstly if you don’t understand RESTful APIs fundamentally, you’re likely to encounter integration issues quite early on. Secondly, unfortunately, if you’re a candidate interviewing for a role and haven’t understood what a RESTful API is properly, you’ll come unstuck in interview.

What RESTful is not (necessarily)

  • A buzzword for a JSON API
  • An API with obscure functionality

What a RESTful API is, and what it has

  • REST is REpresentational State Transfer, RESTful is an adjective, so a RESTful API is an API which subscribes to the REST principles
  • The purpose of REST is to ensure that APIs are easy to understand at a universal level
  • RESTful APIs will have end points which represent entities
  • Those endpoints will respect HTTP methods (also referred to as verbs) to represent the actions you wish to take

Okay, so explain to me the HTTP methods/verbs

  • GET requests represent reading this entity/collection from the data source
  • POST requests represent creating an entity/collection in the data source
  • PUT requests update (destructively, replacing) the entity in the data source
  • PATCH requests update (partially) the entity in the data source
  • DELETE requests remove the entity from the data source

How does that work in practice?

Let’s use the example of a payment provider, you may have entities such as

  • api/customer
  • api/payment
  • api/payment/paymentId/refund

You will always need a specification, but theoretically you know that if you send a POST to endpoint/customer you will create one, if you send a PATCH you’ll partially update that customer. If you send a POST to api/payment you will create a payment, for which you will receive a reference (ID), and if you were to send a POST request to api/payment/id-you-received/refund then you would create a refund against the payment which you specified in the payment ID.

So what’s the point?

The main idea behind RESTful APIs, the same as with other standards such as PSR-X, is to unify the way in which we build APIs, so if I were to say, for example, to an organisation with whom I am going to be integrating “we have a RESTful API you can integrate with” they know, with some level of certainty, how much work there is involved in working with it – they also do not have to have an in depth understanding of my local design, architecture, etc, because the RESTful design abstracts any need for that knowledge.

In summary

RESTful APIs are great, other APIs which are not RESTful can also be great. I just wanted to help inject some clarity on the topic, although, of course, there is plenty of information on the internet around this particular methodology. If one person reads this article and gains an actual understanding of what RESTful is, then it has served its purpose.

Edit: HATEOAS further reading

Valid point raised by DarkTechnocrat on Reddit, for further reading beyond a very basic understanding on REST, you probably should read up on HATEOAS – at a quick glance this article on looks like a good place to start 🙂

General Integration Devleopment

Understanding and Utilising the Expertise in your Project

So this is a topic I thought I would touch on, having worked in lots of environments, on lots of projects, with varying different sets of priorities; I have noticed a few different things; with some very common characteristics across almost all development teams.

The first is that there is a two fold problem when it comes to expertise:

  1. Expertise is never a clear cut, black and white line, every professional, in every role, has a varied set of skills
  2. The second is often the expertise people hold is often overshadowed by personalities on projects

The downside to the second point is that one can end up being entirely neglected as an asset to their project simply because their personality is not overt or extrovert; their voice not strong enough, of their opinions not heard. This can only ever be detrimental to the project, because those skills and expertise are there for a reason.

Project Archetypes

As I say, all projects are different; however, let’s assume a large website build, with complex functionality and integrated partners. I’m making this assumption for the purposes of the most complex project and set of people and expertise possible.

So, let’s look at our archetypes – I’ve given them all abbreviations to try and make this mammoth article slightly shorter.

Please do not be offended by any of the below, these are fictional characters on a fictional project to illustrate a real world point.

The Client (TC)

  • Role: The role of the client is, on behalf of their business, to get the best possible product that fits the requirements, needs, considerations, and budget for that project
  • Strengths & Skills: Nobody knows the inner workings of the business or industry like the client
  • Weaknesses: Will not always understand the implications of their requirements or changes

The Project Manager (PM)

  • Role: To bring the whole project together, encourage collaboration, and efficiency
  • Strengths & Skills: Overall knowledge of the project, from conception to delivery and beyond
  • Weaknesses: Jack of all trades, master of none; often won’t understand the low level detail of the technical elements of a project

The Critical Third Party (CTP)

  • Role: To provide the TC a third party (this could be anything from accounting software to CRM integration)
  • Strengths & Skills: Integrating their software with development teams, understanding their product, offering, and how it adds value to TC
  • Weaknesses: Usually relatively inflexible in terms of integration and capability

The Technical Lead (TL)

  • Role: Ensure the competent and efficient technical delivery of the project (technical development, testing, and delivery)
  • Strengths & Skills: Technical information, planning, and development; and understanding of the easiest route to delivery from a technical standpoint
  • Weaknesses: Sometimes seemingly inflexible, not always empathetic

The Lead Designer (LD)

  • Role: Ensure the deliverable is aesthetically pleasing, on brand, and agreeable by TC
  • Strengths & Skills: Design, brand, understanding and communicating TC’s voice by multimedia means
  • Weaknesses: Where displaying output of integrated data, doesn’t always understand technical implications of design decisions

The UX Expert (UX)

  • Role: Optimise user experience of those using the project, usually with the aim of increasing conversions, a champion of the end user(s) and customer
  • Strengths & Skills: Understanding the impacts on the user of decisions made elsewhere in the project
  • Weaknesses: Can become frustrated at the undervalue UX often receives (and is often brought in late to the project)

The Data and CRM Expert (CRM)

  • Role: Capture as much data as possible for usage by Marketing
  • Strengths & Skills: Understands every single data capturing opportunity
  • Weaknesses: Wants to capture data at every single opportunity, often to the detriment of design and UX

The SEO Consultant (SC)

  • Role: Understand the impact of all technical changes to SEO, advise on SEO strategy
  • Strengths & Skills: Search Engine Optimisation, getting traffic to the website
  • Weaknesses: Very specific skill set, often without wider technical understanding

Collaboration without restriction

As you will have noticed from the above, there are a lot of seemingly key players in this project. Consequently, I have drawn up a couple of pointers which I feel may ease the collaborative process on a project of such scale.

Understand the goals

The project should have a single, clear cut goal. Let’s say it is an ecommerce website complete rebuild and redesign it will likely be something like “increase sales by 25%”, the rebuild having been necessitated by slow page load times due to an outdated database structure, for example.

Now everybody detailed above is going to have their own set of goals. SC will want completely SEO optimised pages, and content population. UX will want a quick route from product to checkout. CRM will want to capture as much data as physically possible. LD will want the design to be beautiful, on brand, and pride of place in his portfolio. TL will want easily changeable code, efficiency and security in their code and servers.

The trick in this stage, is to understand what everyone (not just PM and TC) want out of this project. That way we can predict bumps in the road, and we can work around them before anybody has put pen to paper, code to IDE, or design to Illustrator.

Understand the dependencies

Now we understand what everybody wants and needs, we can see what dependencies there are. Likely dependencies are front end development being blocked until design is signed off. TL is likely to block integrations and development until specifications are signed, but TL is likely to be blocked on a database front by CRM until they have decided exactly what data they want to capture.

In my experience most of this will be blocked, in the first instance, by TC and PM who will need to agree and outline finalised scopes and specifications; once these are signed off the first few parties can get started in their tasks.

Understand the phases

Now we know who is involved, what they want and need, who is blocked and what order things need to be done in. Now we need to draw a plan, the phases, and who will be involved in those phases. There is no sense in a free-for-all as work will need to be redone, and there is no sense in having every party in every meeting. The popular opinion here is that it keeps the project together, working in the right direction. My professional opinion here is that, unless broken down into phases, this is a wasteful usage of time. LD doesn’t need to be in a meeting where the sole concerns are CRM, UX, and TL are discussing how and when to capture data. PM and TC should be actively involved in every phase, however, to ensure consistency across the project.

Stick to the plan

Now we have a plan, most likely with phases, timescales, and tasks down to a granular level. This has been drawn up lovingly and attentively by PM, in conjunction with all parties. On paper it will work. It is now the job of everybody to understand it, they understand the implications of the work they’re doing; so now it’s a case of move forward and don’t look back. That’s what post-go-live is for.

Understand Perfection is a Myth

Perfection is a myth. In digital it’s just not something which is possible. Browsers and devices constantly evolve. As do user journeys and user expectations. If you aim for perfection and refuse to settle for less you will never go live.

Let’s say phase 1 was to understand data capabilities, that’s been and gone. Everything has been discussed, agreed on, and is either in progress or is complete. Don’t revisit those decisions, stick with them. The consequence is we go off plan, which will cause confusion, inefficiency and definite delay; though the cost of that delay will never be crystal clear, as we’re now off plan, and away from estimations.

How you can help the large scale projects you’re on, by following these 5 simple steps

So, from working on lots of projects, here are my top tips for working in big projects.

Step 1 – Voice Your Concerns, Reasonably

There’s always a workflow or a process to follow, if you have concerns, especially those outside of your core skills and expertise; voice them. However, be careful to voice them reasonably. Don’t be deterred if you don’t get the answer you want, but also don’t be unreasonable with your fellow collaborators; their skills are not your skills. Their understanding and expertise is not the same as yours, sometimes, you will have to be told “no”.

Step 2 – Communicate for Value

This point follows upon from something I touched upon in “Understand Perfection is a Myth” above. If you are not adding value, or are not going to add value, you shouldn’t be in the meeting. If you have nothing to say, say nothing.

This does a couple of things, in practice. Firstly it cleans the lines of communications, everybody is only speaking when they have something valuable to contribute. Secondly, it makes it more likely that you’ll be listened to when you do speak, because your track record will be of only speaking when you have a strong and/or valid point.

Step 3 – Understand the People, and the Scope of their Expertise

This is a controversial one, mostly because it assumes people know what their own expertise is. Also because it means each of us as individuals acknowledging what we don’t know. SC may know one or two bits about User Experience design, but nowhere near as much as UX does, therefore it’s not really within SC’s scope to enforce decisions upon UX. Similarly TL knows nothing of brand, so beyond technical consultation should not be enforcing their views on LD during the design phase.

Your expertise is the reason you were brought into the project. If you were brought into the project as the Technical Lead, you should only insist on enforcing your viewpoint on Technical matters, unless explicitly asked otherwise.

Now I understand that this is starting to sound very much “stay in your box”, which is absolutely not the point I am making. The point I am making is when you stop valuing the experts, they lose initiative on their own ground. What that means, for your project, is that maybe TL stops making recommendations about how to keep the page speed down, because SC has made their recommendations which have been listened to over TL (and page speed is TL’s expertise).

It means LD starts doing as he’s told because TC and PM have decided that they’d rather pixel-push the design, than let LD use his area of expertise.

From what I’ve seen, letting people overrule the experts, you end up with a bunch of people capable of making changes, whilst 1 or 2 extroverted figures dictate everything. I’m open to be correct on this, but I’ve never seen it go well.

Step 4 – Empathise & Sympathise, understand Pressure Behaviour

This is the difficult one. Everyone gets a say on everything. Now everyone has to redo all of their work. All the while PM and TC are getting very excitable about how late their project is going to be, or perhaps TC is insisting upon the deadline, which PM knows isn’t necessarily possible.

Now everybody is stressed, everybody is under pressure, and everybody’s behaviours will change. It becomes “every man for himself”, this leads to some really negative behaviours in terms of what’s best for the project.

The first, is inflexibility. “Why should I do that? I’ve done my bit!”. The second, is irritability. “Why are they changing it again? I’m tired of this now!”. The third, is disrespect “Why is UX telling me about my database design, why doesn’t he stick to wireframes!”.

None of these behaviours are important. The bit that is important is that none of them are helpful for creating the best project for TC and PM.

By the time you’ve gotten to this point, it’s too late. Unless there is essentially a blank slate to work from, starting over from the first phase again, the project is doomed to be “just good enough” to get it past PM and TC and go live. Future collaborations (v2.1) will forever be tainted by this project, relationships will be ruined beyond repair. Nobody wins.

How to say “no”

This section is really founded on the principle of understanding the expertise, and respecting it. Sometimes, you do have to say no.

Approach one – “I appreciate your opinion, but I must say from a [insert skill] standpoint, it is not the best way to proceed, because of [reason]”

This approach is asserting that you value the input of other members of the team, but this is within your skillset, and it is a bad idea, which must be justified. If you can’t justify your reasoning, are you being inflexible because you don’t like the idea of someone else contributing?

Approach two – “That’s not viable given the limitation”

This is a great idea! I love it! (Or perhaps, this is a terrible idea, but it’s not viable either way). However, we have to have this done in a week, or the budget is £Xk too small. Value the input you’ve received, but not everything we want to do, is possible. That’s the basis for this particular approach.

Approach three – “That’s not in scope right now”

This is a bit of an extension on the second approach, really. This is very neutral, it’s not a good or a bad idea (unless you want to specify) but the project relies on clearly defined scope, specification, timelines, and such. Adding new things in will always impact deliverables.

This is not the worst thing for TC or PM, it’s the best thing for your current project – which is the one you must concentrate on. Rome wasn’t built in a day, if Social Media Login is a new piece of scope, it’s a great idea, but it has to wait until the completion of this project.

Celebrating the successes

It is so important, when working on a large collaboration like this, that successes are celebrated; and that personal recognition given to the parties responsible. LD had as much to do with database design as TL had with the stunning new logo alterations.

TC and PM can always take a slice of the credit, but if they’re seen to soak up all the glory, it will never be very good for the wider team. Collaboration is about accepting and appreciating the diversity that everyone brings to the table.

This vitally includes celebrating even the small successes, and attributing that credit to where it belongs. Saying “we need a logo” and then taking credit for the new design is akin to having a photo taken with Anthony Joshua, and saying that makes you the second best boxer in the world.

Continuous Collaboration

You now have a successful project under your belts, but it’s digital; it will need revisiting. You have positive foundations to lay your new project on now. There will be a sweeter taste in the mouthes of everyone who has worked on a successful project, with respected and appreciated professionals, where collaboration was free flowing, and so was respect and consideration.

Integration Devleopment

How to build a terrible API

Hi everyone

As you may know, throughout my career I’ve ended up doing a lot of integration development. Middleware, APIs, all that kind of stuff. So I thought I would write a quick list of the things you should definitely do to make your API really terrible for other developers to use, so that they put their heads through their desks trying to integrate with it.

Judging from some of the APIs I’ve had the misfortune of integrating with, I thought this was worth writing, for anyone who is building an API.

Obviously this post is humorous or satirical, if you like. Please, for the love of all that is good and sacred in this world, don’t follow these tips!

Tip #1 – Don’t write a darn thing down

No developer likes reading documentation, let alone writing it. So the best tip to make sure that everyone hates your API is to make sure you don’t write a single thing down. Not a thing, nada. Let them guess at how to use it and marvel at your ivory towered genius.

Tip #2 – Make sure that your API is totally inconsistent, especially with documentation

So if you did go to the effort of cobbling together some documentation, and this is critically important, you absolutely must ensure that it is totally out of date. It should be at least 2 major versions behind your developments to keep your job safe.

Also, make sure there are plenty of typos in field names, don’t spell check anything at all; and be sure to use plenty of three letter acronyms that only people within your organisation understand. This will make integration even more surprising.

Top tips for inconsistency include ensuring that you use an absolute minimum of three naming conventions, and at least four different response structures.

Tip #3 – Make sure you use all the wrong terminology

If your API is a straight-up JSON-POST API, which is really simple to use, make sure you throw in a few curve balls. Maybe some GET requests in there, maybe even the occasional PUT or PATCH. It just keeps things entertaining. Parse the API key in the header in some requests, a query string in others and part of the initial request object in the rest.

Also, make sure your account/client/public facing departments refer to your RESTful APIs as Fairy Liquid, and your SOAP APIs as Sleepy ones. This will make sure everyone finds your technical capacity fantastic and will marvel at your technological prowess.

Tip #4 – No version control

If you don’t make sure that your API is versioned, it will keep the developers integrating with it in work; because every time you change it they will have to rapidly drop all of their mission-critical developments to fix any integrations. Be sure to release a patch, break some calls, and then roll it back. Just to punish anybody not using version control on their integrations.

Don’t do anything silly like adding version control, or advanced warning of changes, and definitely don’t even think about doing Beta releases. That just takes the fun out of integration development.

Tip #5 – Throw as many random errors as possible

What you definitely shouldn’t do, under any circumstances, is have a unified and consistent way of handling errors, such as a resource not found or an API key not matching. These errors must be handled differently for every single possible error, on every single possible call.

This is really good for the economy, because it ensures that any developers integrating with your API will have to test every single possibility, against every single call you have. This will keep them in work. You’re doing them a favour, honest.

Why give a JSON error response back, detailing an error code and a human readable message, when a 404 or a 500 will do, that’s what they’re there for! Bonus points if you can throw a 301.

And you’re done.

If you follow these 5 simple rules, you will have a super-awesome-epic API built our of pure ivory, which people will have to pay you to integrate with, because nobody else will know how to.

Then you’ll become like Zuckerberg-cum-Gates rich and be proper leet hacker wizard man (or witch woman).

So there you have it. From someone who’s built, and integrated with, several (probably dozens of) APIs. The top list of things you should definitely never, ever, ever do, if you don’t want to burn in hell for the rest of eternity, stuck in an infinite loop of torture, ridicule and unknown fatal exceptions.

Peace out y’all! 🙂

I’d genuinely love to hear any additional tips you’ve got to build brilliant APIs, and your hilarious anecdotes which are only funny now because you’ve finished the course of therapy and sedatives and recovered from your caffeine addiction enough to laugh at them!