ScaleScaleScaleScale

Great Architectures, Stacks & DevOps at Webscale

By Chris Ueland


Scaling News from Around the Web – 21st Dec 2015

Interesting thought:

No one at Google has the title of Architect. There is no central approval for technology decisions.  Most technology decisions are made by individual teams locally for their own purposes, they are not made globally.

News

  • Slack has grown to 2 million users in just over 1 year. They’re now launching a 3rd party App Directory and an $80 million fund to encourage developers to write Slack compatible apps. Read more on the Slack blog
  • Lessons learned from Google, Ebay, Amazon and Twitter. Their architecture has evolved into something similar: a set of polyglot microservices. Built with different languages, almost seemly without a plan or direction

Machine Learning

  • Facebook Open Sources their BigSur AI hardware. Build around the  Nvidia Tesla M40 GPU, with up to eight Nvidia Tesla M40 per Big Sur chassis. Each Nvidia Telsa M40 card has 3072 cores and 12GB of memory.
    13X Faster Training

Network

  • GCHQ, the UK’s communication’s spy agency have Open Sourced their Gaffer database. Based on Graph and now under the Apache 2.0 license, the system is designed to analyze relationships between different pieces of data. This could in theory be used by GCHQ to find out who or what is at the center of a network.
  • Google’s moving towards Software-Defined Network (SDN) infrastructure in order to curb costs on the worlds largest private WAN. In fact, Google’s private facing WAN accounts for 10% of all global internet traffic. By effectively building thier own network hardware, adding a layer of abstraction they are able to remove vendor lock-in, reduce costs and increase control.

Tools

  • Ceph – An open source distributed block and file storage platform. A viable alternative to high end SAN storage arrays.
  • Q – Run SQL-like queries directly on CSV files
  • Git-appraise is a git plugin which implements a code review workflow for distributed teams, storing the review data in the git repository itself.
  • Gaffer a database system designed to analyze relationships between different pieces of data.
profile

John Cattrall