ScaleScaleScaleScale

Great Architectures, Stacks & DevOps at Webscale

By Chris Ueland


Backend Teardown: Muut

Muut’s Stack

Compute
Compute: EC2 (Reserved)
Cost Management: Cloudability
Config Management: Chef
Content Delivery & Storage
CDN: MaxCDN
Storage: S3
Data Store
Cache: Redis
NoSQL: MongoDB
Distributed DBMS: Cassandra
DNS
Provider: DNSMadeEasy
Monitoring
Uptime: Pingdom
Server Monitoring: New Relic
API Performance: Nodetime
Application
Notifications: Mandrill
Transactional: Amazon SES
Search index: Solr
Network
Load Balancer: HaProxy

profile

The guys at Muut have architected a next-generation real time commenting platform. It’s strikingly fast and well architected. Courtney has been a friend for many years and Muut has been using MaxCDN since the first version of EdgeRules. This ScaleScale Q & A is the first in a series focussed digging into the details of well designed backends to give you ideas and share information.

–Chris / ScaleScale / MaxCDN

What is Muut?

Muut is an embeddable commenting and forum system. It’s easily extended via JavaScript and fully styleable with CSS.

Using Muut you can build an onsite community without having to resort to separate servers or a completely separate site. This means even if your site is fully static and served 100% from your content delivery network you can still have a fully dynamic yet completely integrated community that fits seamlessly with your site.

Our free service gives you unlimited traffic, posts, users, and uploads.

Muut co-founder Courtney Couch

Muut co-founder Courtney Couch

What are your app and infrastructure design methodologies?

We’re a completely test driven development shop. We write tests for everything from our functional requirements to our scaling and performance requirements before any changes are made.

Performance and automation are core values for everything we develop. Our actual workflow is sort of something specific to us and always changing. In fact we have a channel on Slack dedicated solely to workflow changes and improvements.

How is it different than things like Disqus?

Disqus is a free ad driven product running in an iframe.

Muut running very much like a jQuery plugin which means that it can be extended, and since it’s not in an iframe this means it can be fully styled simply with CSS.

Muut is also a full community platform with commenting and forums not simply a commenting system. The fact that these come from the same system is a key differentiator (same users, same login and same UI).. With items like our membership only functionality, webhooks, and so on, users have the ability to build totally customized communities deeply integrated into their existing systems.

Aside from the deep integration capabilities, Muut is incredibly simple to use, our designer makes it easy for anyone to really customize how our product looks, and our integrations into existing platforms continue to grow deeper and easier to use.

What is your stack? Software? DBs? Servers? CDNs?

We are running on a pretty diverse stack.

Our source of truth (primary data store) is on Redis. Our API servers are all nodejs.

Aside from that we have servers running MongoDB, some Python, Ruby, Java, Cassandra.

Despite the variety of datastores and languages being used we are not currently using a relational database.

Our servers are all in AWS and we heavily leverage MaxCDN for our static content. Our application directly talks to MaxCDN to invalidate files that are updated when settings change or files (SEO content) get’s updated.

As far as infrastructure we rely on Chef + Docker to automate and handle deployments.

Muut Architecture

Muut Architecture

Muut runs on a pretty diverse stack running on AWS and using MaxCDN for static content

What are some of your numbers now and what are you built out for?

We’re pushing around 300m application initializations per month right now and our test environment emulates several billion. We’re at around 5% utilization of our current network (peaking up to about 10%) and we built our architecture to scale horizontally so spinning up a few new app servers or data shards is all we need to do if we start getting too loaded.

What have you learned from outages?

There’s no such thing as a ‘non critical’ service.

We considered different internal processes non-critical before and found out that outage of a non-critical service can have unexpected impact on critical services. We now consider every process critical whether it’s just a housekeeping process or a load balancer.

How do you measure uptime?

We use a combination of New Relic monitoring and Pingdom hitting our API, testing DNS, ensuring processes are running and responding as necessary.

How do you measure performance?

All API calls are logged with the performance time into Redis then summary data is pushed over to Cassandra for long term archival.

We also heavily rely on the reporting from New Relic on every piece of our infrastructure. All API responses also include server performance data in case any curious users want to know. We provide performance data on every API call down to microsecond precision.

What do you use for source control? project management? communication?

We use quite a few different tools for different aspects of our development.

All projects follow this process:
Start out as a Trello card Becomes a Balsamiq mockup Becomes a Github repo

Slack is for our basic day to day communication.

We have always shied away from the really feature rich project management tools people end up using chat or email instead, or simply get caught up “managing” rather than coding.

How important is infrastructure cost & efficiency for Muut?

Cost to performance is a huge factor for us. Our free service gives users unlimited traffic, posts, users, even image uploads and we can’t subsidize that with advertising since we aren’t an ad driven service. This means our infrastructure has to be incredibly efficient. Performance for us means cost efficiency. Dropping API response times from 5ms to 4ms for example won’t impact users but that means 25% more capacity on the same hardware. Users win with a service that’s lighting fast, and we win by keeping costs incredibly low.

What’s next for your infrastructure?

We’ve still got some legacy services not yet migrated over to Docker so we need to complete that transition. We’re then planning on moving some critical parts of our architecture to the edge.

Initially we want to simply move our load balancers to the edge and setup some websocket termination there so that the connection setup and SSL handshakes only go to edges. Ultimately as much as we can push to the edge we will.

#Protips

  1. We use MaxCDN to do full site w/ EdgeRules. Give us control over the backends.
  2. We use DNSMadeEasy to do an ANAME lookup so muut.com responds without www.
  3. To keep costs lower we use 100% heavy reserved instances and leverage cloudability for monitoring costs.
  4. Having a lean project management stack allows us to code more.

Popular search terms:

  • 469Q
  • 80CB
  • agree6op
  • can41d
profile

Chris Ueland

http://www.ueland.com

Wanting to call out all the good stuff when it comes to scaling, Chris Ueland created this blog, ScaleScale.