Technology Stack

Why?

As a team of people using technology to make digital products,
it's essential to be unambiguous
about the technology/tools we use,
so that everyone is clear what we all need to learn/use
to make product(s) that are functional, fast, beautiful, useable and reliable!

If anything is unclear or you have any questions please
ask.

What?

This document + diagram describes the full "PETE" Technology Stack
we use for dwyl products and services (clients).

Each element in our stack was carefully selected based
on its individual merits.
When assembled into a seamless "machine",
the stack is unrivaled for developer productivity
and world-class quality!

The "PETE" Stack

"PETE" is an acronym1 for the following elements:

• Phoenix is a Web Application Framework that does not compromise
  on speed, reliability or maintainability!
  Phoenix is the "successor"
  to the incredibly popular "Ruby-on-Rails" framework.
  Built from scratch by highly experienced engineers
  who worked on/with Rails. It solves
  all of the speed/socket/scaling/concurrency, issues
  people felt when building/using Rails apps.
  The list of benefits Phoenix has over
  (virtually every) other Web Frameworks is extensive.
  Please see:
  github.com/dwyl/learn-phoenix-framework#our-top-10-reasons-why-phoenix

• Elixir is the functional programming language used
  by the Phoenix framework. Elixir is a beautiful language
  written from scratch to be friendly, concise and efficient.
  Yes, Elixir not as "popular" as JavaScript, Java, C# or PHP,
  but the adoption is growing rapidly and most importantly
  many experienced developers are gravitating towards and
  describing it as their "most wanted"
  Also a language's popularity has more
  to do with the intellectual inertia people/companies have because
  they allow existing (legacy) codebases to dictate future development;
  i.e.
  sunk cost bias.
  see: github.com/dwyl/learn-elixir#key-advantages

• Tachyons is the most sane way of creating a beautiful web app UI
  that can easily be extended by a team of people
  without fear of one person's change "breaking" another feature.
  Unlike "traditional" CSS which - as it's name implies - encourages
  "cascading" of styles, Tachyons makes the style of each component specific
  and local to that component.
  see: https://github.com/dwyl/learn-tachyons

• Elm is a front-end developer's dream
  made reality by a community of unbelievably
  brilliant and dedicated developers.
  While Elm is
  incredibly fast
  the reason we love elm has nothing to do
  with "benchmarks" and everything to do
  with the experience of writing/reading/maintaining
  an app built with elm; that experience is like
  a reassuring hug from a loved one.
  When you discover elm, things that used
  to be difficult or "error-prone" become easy and reliable.
  See: github.com/dwyl/learn-elm#why

Elm has considerably better performance
than similar/alternative front-end frameworks
(e.g: Angular, Ember, React, Vue.js etc.)
so hopefully the "performance" discussion can "go away"!

¹Thanks to Jimmy Ruts
for coining the "PETE" acronym. He's the guy we go to for naming
and other "hard things!"
;-)

Database?

The reason we do not specify our Database in the "PETE" Acronym is
that Phoenix allows us to use any Relational Database.

By abstracting the data layer using "Ecto" the application is "decoupled"
from the database.
This means that if a client asks us to deploy to MySQL or
Microsoft SQL Server
(e.g. because they already have in-house capability
for maintaining one of these databases)
we can easily accommodate that without re-writing any of the Phoenix app!

We Prefer PostgreSQL (Postgres)

Our "standard" (preference) @dwyl is for Postgres.
see: github.com/dwyl/learn-postgresql

Postgres is the most "mature" Open Source Relational Database.
It's 100% Free (including all "advanced" features)
and has been deployed and battle-tested in every environment
from AWS to "Bare Metal" and Google Cloud to Microsoft Azure!

Many well-known/successful apps rely
on Postgres as their main database.
NOT that you should adopt a particular technology
based on who else is using it,
but it's good to know that plenty of teams
are getting excellent results with PostgreSQL!

See: List of Organizations Using PostgreSQL

We have used most of the "popular" Relational Databases.
e.g: MySQL, MS SQL Server, Oracle and Aurora; all
RDBMS
have their pros/cons.
The reason we like/use Postgres is because the community
is superb. There is a great "bank" of answered questions on
StackOverflow
and new questions get answered fast.

Operating System?

A "traditional" LAMP stack includes the Linux Operating System
in the name.
The "PETE" stack runs on any (desktop/server) Operating System
and can be deployed to any "cloud" infrastructure provider.

While we have a strong preference for Unix (e.g. FreeBSD) or
Linux (e.g. Ubuntu or CentOS) we know that
both Phoenix and Postgres run
on almost any environment including Microsoft Windows Desktop & Server.

Continuous Integration

We use and recommend Travis-CI for Continuous Integration (CI).
If you or anyone on your team are new to Travis-CI,
please checkout our complete beginner's guide:
github.com/dwyl/learn-travis

Deployment

We make a point of deploying our work as soon as
there is something worth showing to the target audience of "end users"
so that we can get feedback as early as possible.

"Development" and Testing

We use Heroku for all our development deployment.
It's by far the most time-effective way to "ship"
the project and get it tried & tested by real people.
Using heroku is like having a "Dedicated DevOps" person
on the team.
Once the deployment is configured we can automatically deploy
a new version of app for each feature and test incrementally.
This alone is worth the modest fee Heroku charges for their service!

If you are completely new to Heroku you can get started
in 10 minutes with our
complete beginner's tutorial: https://github.com/dwyl/learn-heroku

For a step-by-step guide to deploying a Phoenix ("PETE")
application to Heroku,
please see:
github.com/dwyl/learn-phoenix-framework/heroku-deployment
(special thanks to Tony & Jimmy for doing a superb job putting the guide together!)

"Under the Hood" Heroku is using AWS for their infrastructure.
So when we need to "graduate" an app from Heroku to AWS it's easy!

"Production"

Our preference is to deploy to Amazon Web Services (AWS)
because we have good experience/knowledge of the platform.

Microsoft Azure

One of our projects has a requirement to be deployed
to Microsoft Azure Cloud. We achieve this with the following
configuration:

Application Server

The Phoenix Application Server is hosted on (a minimum of)
Two Linux Servers
(often many more which "message"
one another to distribute load as a cluster).
The "cluster" is managed by Erlang's "Supervisor".
The Erlang Supervisor is the "Gold Standard" in infrastructure management,
having been used by Telecoms companies for over 20 years in production
with some Telcos reporting 99.99999% ("seven nines") of "up-time".

It's far more likely that the infrastructure provider (e.g. AWS/Azure)
will have a fault in their network/datacenter than an Erlang server "crashing".

Database Cluster

Thanks to Esmaeil Sarabadani of Haufe.com there is a "Template"
for deploying a Highly-Available PostgreSQL Cluster on Azure
this can be setup with only a few clicks and scaled automatically
to as little or large as required!
See: http://dev.haufe.com/PostgreSQL-Cluster-Azure/

SSL/TLS Encryption

All communication is over secure/encrypted channel
(by default at all times)
to protect the data/privacy
of people using the applications we make.

We recommend using the "Let's Encrypt" service for SSL Certificates
it's 100% Free (and provided by a Non-Profit foundation)
to help you get started, we wrote a
step-by-step setup guide for apps deployed to Heroku:
SSL-certificate-step-by-step-setup-instructions.md

Beginner Tutorials?

We have crafted a "Complete Beginner's Guide"
for each element in the stack, so that we:

• Document our collective learning
  while we are building projects.
  (because as humans we forget fast unless we write it!)
• Share our knowledge with other people so we can
  • Help to train (potential) new team members
    as quickly/effectively as possible.
  • Collectively iterate on our knowledge and "level-up" as a team!
  • "Onboard" the client team (who may want/need) to
    support/maintain the codebase/project if/when we seamlessly "hand over".
  • Inform the wider community of both technical and non-technical
    people ("stake holders") who are generally
    interested in understanding the project.
  • Enlighten other teams/organisations/agencies/etc. we aren't in
    direct contact with that there is a "more fun" way of building software!
• Make everyone's life easier/better
  by having a "launch pad" for
  rapid learning!

For an example project where this "Stack" is implemented
see: https://github.com/healthlocker/healthlocker

Node.js Stack

We have deployed our Node.js stack for many clients
and internal apps and achieved great results!
It works really well and we have not had any issues with "performance"
or "scaling" deploying to AWS.

For examples of apps built using our Node.js Stack
see: https://github.com/TheScienceMuseum/collectionsonline
and https://github.com/emfoundation/ce100-app

There's no "reason" to "rewrite"
any of our existing projects to any other "stack".
Node.js works perfectly well and will continue to be supported for the lifetime
of the project(s)

See: "tl;dr" section below if you are interested
in why we decided to "evolve" to "PETE" for new projects...

Overview

The following diagram is an overview of our Node.js stack:

Note: To edit/improve this diagram: https://github.com/dwyl/technology-stack/issues/1

We have produced a complete beginners guide for each of the components in our stack. (see below)

Open Source Projects We Use

For Us By Us

We craft code to scratch our own itch and everything we do is always Open Source, Project, Used For, Build Status, Test Coverage, Dependency Status, Tutorial, --------, ----------, :-----:, :--------:, :------------:, -------, env2, Loading Environment Variables, , , , learn-environment-variables, esta, ElasticSearch CRUD, , , , learn-elasticsearch, goodparts, Consistent Code (Linting & Style), , , , goodparts#why, hapi-auth-jwt2, Authentication & Sessions, , , , learn-json-web-tokens, hapi-error, Human-Friendly Error Messages, , , , hapi-error#why, hapi-login, User Login, , , , learn-hapi, hapi-postgres-connection, Postgres Connection Pooling, , , , learn-postgresql, hapi-redis-connection, Simplify Redis Connection, , , , learn-redis, hapi-register, User Registration, , , , learn-hapi, hapi-riot, Server-side (Fast) Rendering of Riot Tags, , , , learn-riot, > Note: to update this table, use the script: generate_dependency_table.js

Dependencies (projects built by people we trust)

• Node.js - the most popular JavaScript runtime for easily building fast,
  scalable simple network applications. Lightweight and efficient, perfect for
  data-intensive real-time apps. http://nodejs.org/
• Hapi.js - A rich web framework for building applications and services.
  https://github.com/dwyl/learn-hapi
• Socket.io - a JavaScript library for realtime web applications.
  It enables realtime, bi-directional communication between web clients and
  server. Socket.io lets us send data to/from everyone connected to our app(s)
  without having to refresh the web page. http://socket.io/
• Riot.js - is the most light-weight user-interface (UI) framework
  available which is compatible with IE 8/9 and has good
  server-side rendering (which means pages load faster for slow devices like budget smart phones).
  see: https://github.com/dwyl/learn-riot
• Redis - the most popular in-memory data store which is essential
  for building the fastest possible apps.
  read more: https://github.com/dwyl/learn-redis
• ElasticSearch - the most feature-rich search engine. we use
  it to find things fast. Learn more: https://github.com/dwyl/learn-elasticsearch

Development Dependencies

We carefully select and only use well-maintained "pure" JavaScript modules
in our development toolchain:

• Tape for testing: https://github.com/dwyl/learn-tape
• Istanbul for Code Coverage: https://github.com/dwyl/learn-istanbul
• Pre-commit for ensuring all commits pass strict quality checks
  before being pushed to GitHub. see: https://github.com/dwyl/learn-pre-commit
• GoodParts "linting" and checking code style is consistent:
  https://github.com/dwyl/goodparts
• CodeCov for detailed test/code coverage stats
  https://github.com/dwyl/learn-istanbul#tracking-coverage-as-a-service
• CodeClimate for tracking code quality:
  https://github.com/dwyl/learn-codeclimate

tl;dr

There is no shortage of options available for
Technology Stack!
See: https://www.google.com/search?q=technology+stack&tbm=isch
So, how did we arrive at the conclusion that "PETE"
was "the one" for us...?
We already had a really good
Node.js Stack
which worked well for us and our clients. so . . .

Why Try a "New Stack"?

Why Try Something New When We're Already Good with the "Old"...?

Our reasoning for
considering an alternative approach/stack for building web apps
was fueled by our
curiousity
and
"shoshin".

"The important thing is not to stop questioning. Curiosity has its own reason for existence." ~ Albert Einstein ~
"Old Man's Advice to Youth: 'Never Lose a Holy Curiosity.'"
LIFE Magazine (2 May 1955) p. 64”

In November 2016 we (once again) questioned our assumptions,
re-examined and
surveyed
the "landscape" of "emerging trends" in web app development.
We were pleasantly surprised delighted to see the amazing progress
made by the people in the Elixir, Phoenix and Elm communities!
These technologies are set to "take off" in 2017 and we are excited
to be sharing the technical/competitive advantage with our clients!

Please see: https://github.com/dwyl/learn-phoenix-framework#questions

Making Difficult Decisions

One of the most "difficult decisions" you will make in your "career" is
which technologies and tools you will use
to deliver the desired solution/benefit to the "end users".

Most people have the Tech/Tools decision made for them
by the company/organisation/boss they work for
(e.g: Java -> Spring, Ruby -> Rails or PHP -> WordPress or Symphony, etc.)
This is because most companies already have an existing app in "production",
which you have been hired to extend.

Occasionally you will get the chance to build an app from "scratch"
however most of the time someone else (the "Architect")
will make the decision for what "stack" to use on your behalf,
so you will still be stuck with someone else's choices.
If you are incredibly lucky the "Architect(s)"
will have done their homework: surveyed the latest industry
trends and investigated the "new and promising" technologies
e.g: Stack Overflow
"Most Wanted" list.

Most "Application Architects" will pick one of these 3 options:

1. Go with what you (already) know, use existing stack
   with a minor variation because it's "easy to deploy" with
   the existing infrastructure and will not get questioned by the "Executives",
   DevOps team or "Compliance" department. This is the easy choice
   and nobody ever got "fired" for sticking with what they know "works".

2. Buy the whizz-bang all-in-one solution sold to them by the "Consultant"
   from "Big Vendor XYZ" (outsource the thinking to a sales person who last
   wrote code in the 90's ... seems like a great idea ... NOT!)

3. Be "Bold" and try "Popular Framework XYZ" and hire an external
   team to build the new magic app. Then attempt to "up-skill" the internal
   team to maintain the code written by the consultants.

None of these choices is optimal, all have different levels of risk/reward.
The "hardest" choice to make is the one where you try something
totally different
The reality is that very few people
have the time/resources/mindset/inclination to take a step back
and open their minds to the idea that there might be a "better tool"
for the job than the one they are currently using.

Toast Knife Analogy

Imagine Want to Make Yourself Some Toast.
Let's write a quick "user story" for this:

As a peckish person
I want a slice of tasty toast
So that I can have some crunch for my brunch!

(let's assume that you bought a loaf of bread from a baker
to reduce the options for solutions for simplicity
(i.e. not baking from scratch!)

The "traditional" way to "solve" the challenge of making toast:

• Cut bread with bread knife
• Put sliced bread in toaster
• Turn on toaster for pre-determined amount of time
• Wait patiently for toaster to make toast

But ... what if instead the "old" way we just described,
someone invented a way to toast the bread while slicing it...?!

Simply by using the "New Tool" - in this case the
"Toast Knife" - you can
simplify the process to a single step!
This is the power of being open to considering "New" Tools/Technologies!
Get the same result in fewer than half the "steps"!

Focussing on Long-term Benefits

The short-term cost of learning a new stack
(programming language or framework) is time,
We contend that the 1-3 week learning time (depending on the focus of learners)
will pay for itself within 3 months
(often sooner if the team is large/distributed because the structure
offered by Phoenix means everyone working on the project
has a disciplined approach to adding new features)

Further Reading on Long-term Thinking

• https://hbr.org/2012/08/thinking-long-term-in-a-short
• https://hbr.org/2011/03/capitalism-for-the-long-term

Contextualising Technology Adoption (Mini History Lesson)

In 1996 Nokia introduced the
"Communicator"
and was a incredible revolutionary innovation!
Internet, Email and Fax in your Pocket!!

Nokia continued to dominate the mobile phone industry/market for the next
decade producing the best-selling
5110 and 3310 we all remember!
But by being "ahead" Nokia was unable to see the "contender"
coming "out of nowhere" to challenge their position.

In 2006 nobody was making/buying "smart" mobile phones
with glass touch screens that ran "apps" ...
in January 2007 Steve Jobs introduced the iPhone
and literally changed the industry!

The dominant/incumbent mobile phone maker Nokia had
49% market share in 2007
mocked Apple's lack of features, poor battery life and high price.
By 2013 Nokia had 3% Market Share (for new device sales) and was sold off "for parts" to Microsoft
while Apple was the most valuable company
in history!

Many people still buy "feature phones"
(the polite name for a device that does not have any
"smart" functionality!)
but few people can convincingly argue that the reason
they don't have a smart phone is because they don't want one;
over 90% 16-24 year olds own a smart phone and that number is expected to reach 100% by 2020 ...
ask someone in their 20's if they would go "back"
to using a non-smart phone
and see what they say ... "No Way!!"
they laugh uncomfortably and
admit that: "My Smartphone is my Life!"
We feel exactly the same about "old technology".
Sure the "old way still works",
but if you can inexpensively switch
to something demonstrably better in every aspect,
why would you stick with the "feature phone" of web frameworks...?
It's like taking the Bus to work when there's a perfectly
good teleporter right next to the bus stop!! Madness.

We are not suggesting that everyone
is going to suddenly flock to the "PETE" stack
the way people adopted smart phones.
This is merely an illustration that when a technology
has a specific/measurable advantage to it's users
the adoption can be fast.

In the case of programming languages or application frameworks,
moving one framework to another is a much more difficult decision.

But one thing is for sure we are going to use the "smart phone"
even if other people insist on using the "brick".

But Phoenix Uses Node.js for Static Asset Compilation ...

Yes, if you are using the (http://brunch.io/)
"static asset compilation".
We aren't using it for our project(s) because our only
"static assets" are the Elm files which we compile using elm-compiler.
The elm compiler
is written in Haskell so technically we aren't writing any JS
in our PETE projects.

Does it Scale?!?

If you are new to web development,
please focus on UX and forget about "scale"!

Unless you work somewhere that
already has "millions of users" and
your team cannot consider anything that
does not support a million concurrent connections...!

But let's face it, most people have imaginary scaling
issues
not real ones.
discussing "scalability" before you have 10,000 paying customers is
a waste of time!!

Stop worrying about "scalability"
and instead focus on building something useful
focus on User Experience not "backend" scalability!

The good news is that Phoenix "scales" really well!
see: http://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections

Forget about "scaling" until you have made
something people want
and are paying for!
Then use the pile of cash you got from your product
to hire "engineers" to make it available to more people!!

No JavaScript in "PETE"...?

This not the place to "diss" JavaScript;
plenty of people have written blog posts/tweets "ranting"
about the
"State of Web Development".

e.g: Douglas Crockford (the authority on JS) recently gave a presentation on
The Post JavaScript Apocalypse: https://youtu.be/6Fg3Aj9GzNw
(in which he describes all the "features" of JS that are "unnecessary"...)

Or Max Ogden (a prominent JS developer) who created
a website dedicated to the "Callback Hell" issue: http://callbackhell.com
why do we even need to have an expression for that?
why can't elegant asynchronous control flow
be the only way to write code?

The fact is: we only use JavaScript because it is the
"Lingua Franca"
that all web browsers "understand".
It's definitely not because it's a
"better" language than Python, Julia or Lisp;
we write JS out of necessity not choice.

With Elm we no longer need to write our Client-side
(progressive enhancement) code in JavaScript!
We can write in a beautiful/functional language
and "compile" it to JS for running in Browsers.
The JS that is produced by the Elm compiler
is almost always more efficient/faster
than "hand-written" JS,
so why would we waste our time with writing JavaScript...?!

If you aren't .ready() to try something (way) more productive
than JavaScript, .then please just ignore this!
We have written this "Technology Stack" description for ourselves
and not because we want to "convince" anyone.
But ... we wanted to put down our thoughts
in case anyone is
"on the fence"!
If you're in any doubt, Just Do it.

What About Full Stack JavaScript?

We still think that "Full Stack JavaScript"
is a compelling proposition
especially for people who are just starting out!
If you are learning programming and want a "quick start"
try Meteor.js!!

Alternative Databases?

If we were to consider an alternative to SQL, we
would use RethinkDB:
https://rethinkdb.com
But we are relieved that the Phoenix team
is focussed on PostgreSQL because that eliminates
the "ambiguity" or "discussion" of "which database" to use!
Postgres is a fantastic "general purpose"
store that has a rich ("structured") query language
that lets you JOIN data!!
Also, now that Citus DB is Open Source
we know that Postgres can easily handle billions of writes per day!