ScaleScaleScaleScale

Great Architectures, Stacks & DevOps at Webscale

By Chris Ueland


Scaling News from Around the Web – Dec 2015

Interesting thought:

Machine intelligence experts predict that computers will gain human-level ability around the year 2050, and superhuman ability less than 30 years after.

News

  • Ebay’s Kylin Distributed Petabyte-Scale Distributed Analytics Engine is now an Apache Top Level Open Source Project.
    Last year Ebay said “Our largest use case is the analysis of 12+ billion source records generating 14+ TB cubes. Its 90% query latency is less than 5 seconds”
  • Zync Cloud Rendering is out of beta, scalable 3D rendering available to home users through to industry rendering studios.
  • Google says it’s $10 million quantum computer is more than 100 million times faster than a regular computer chip. Tasks such as image recognition, detecting exoplanets, weather forecasting as well as the ability to analyze DNA sequences could be advanced greatly.
  • High Quality Video Encoding at Scale – read more on the Netflix TechBlog

Machine Learning

  • Open AI – hello world – Elon Musk backs $1bn non-profit AI research company.“Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return,”
  • Facebook to open-source AI hardware design FAIR for Faster, more versatile, and efficient neural network training.
  • Torch is a scientific computing framework with wide support for machine learning algorithms. It is easy to use and efficient, thanks to an easy and fast scripting language, LuaJIT, and an underlying C/CUDA implementation. Get the latest version on Github
Open Rack V2 compatible 8-GPU server
Open Rack V2 compatible 8-GPU server

Network

  • ELS – Latency Based Load Balancing . Great performance, almost as good as JSQ in benchmarks with non-failing machines. Outperforms JSQ when machines fail. Read more on the Spotify Labs Blog
  • Streaming video and audio traffic now accounts for over 70% of North American downstream traffic in the peak evening hours on fixed access networks. Five years ago it accounted for less than 35% – Read more on Sandvine.com
    Sandvine

Tools

  • Nchan is a scalable, flexible pub/sub server for the modern web, built as a module for the Nginx web server.
  • Dgraph.io is a scalable distributed low latency high throughput Graph database. Designed for efficient retrieval, offering low enough latency to be serving real time user queries, over terabytes of structured data from multiple machines or datacenters.
profile

John Cattrall