Just in time for EmberConf next week, we’re excited to announce a new version of our EmberFire bindings, along with an example blogging app to showcase the magic of using Firebase with Ember.
Our new Ember bindings now work directly with Ember Data. This means you can now use Ember Data’s relationships with Firebase to define hasMany and belongsTo relationships between your models. If you’re new to Ember Data, Ember’s guide to models is a great place to start.
With these updates, EmberFire now works only with Ember Data. For Ember developers who aren’t using Ember Data, Michael Jackson has written ember-firebase - an awesome, thoroughly tested set of Firebase bindings for Ember. If you’d still like to use the previous version of EmberFire, it is available in the v0.1.0 release on GitHub.
With our new bindings, you can start using EmberFire in just two steps. First, you’ll need to include the necessary libraries:
<head> <!-- Ember + Ember Data --> <script src="http://builds.emberjs.com/canary/ember.min.js"></script> <script src="http://builds.emberjs.com/canary/ember-data.min.js"></script> <!-- Firebase --> <script src="https://cdn.firebase.com/js/client/1.0.17/firebase.js"></script> <!-- EmberFire --> <script src="https://cdn.firebase.com/libs/emberfire/1.0.13/emberfire.min.js"></script> </head>
You can also download EmberFire and all its dependencies from Bower via bower install emberfire --save.
bower install emberfire --save
Then, simply create an instance of the DS.FirebaseAdapter in your app and add your Firebase URL:
DS.FirebaseAdapter
App.ApplicationAdapter = DS.FirebaseAdapter.extend({ firebase: new Firebase('https://<my-firebase>.firebaseio.com') });
With the adapter set up, you can now interact with the data store as you normally would with Ember. For example, calling find() with a specific ID will retrieve that record from Firebase. It will also start watching for updates and will update the data store automatically whenever anything is added or removed.
To help you understand our new EmberFire bindings we’ve created FireBlog, a realtime blogging app with posts, comments, and users. We’ll walk through the core concepts here, and the full code is available on GitHub.
Because both Ember Data and Firebase work best with denormalized data, FireBlog does not embed comment data directly in each post. Instead, each post has a comments object with a comment ID pointing to the comment data in /comments. We’ve created three top-level keys for posts, comments, and users, and the data structure in Firebase looks like this:
/comments
Each post in our blog will have a title, body, published, user, and comments attributes. Using Ember Data’s relationships, each post belongsTo a user and hasMany comments. We’ll use Ember’s computed properties to format the date to display on the blog post.
App.Post = DS.Model.extend({ title: DS.attr('string'), body: DS.attr('string'), published: DS.attr('number'), publishedDate: function() { return moment(this.get('published')).format('MMMM Do, YYYY'); }.property('published'), user: DS.belongsTo('user', { async: true }), comments: DS.hasMany('comment', { async: true }) });
The comment and user models are set up in a similar way, which you can see on GitHub.
To get all blog posts from Firebase we can use the standard Ember Data way of fetching posts, like the following:
App.PostsIndexRoute = Ember.Route.extend({ model: function() { return this.store.findAll('post'); } });
This will fetch all posts when the page first loads, and will also listen for updates to Firebase. Whenever posts are added, the data store will update automatically and new posts will be displayed in the DOM.
Similar to how we fetched all posts using store.findAll(), we’ll fetch individual posts using store.find():
store.findAll()
store.find()
App.PostRoute = Ember.Route.extend({ model: function(params) { return this.store.find('post', params.post_id); } });
We'll use the following template to display a single post. Then we'll create an Ember.Component to display individual blog posts. We'll pass in the model from the route (post=model) as a property to show the post's title, author, date, and comments:
Ember.Component
<script type="text/x-handlebars" data-template-name="post"> {{ "{"}}{{ "{"}}fire-post post=model onPublishComment="publishComment"}} </script>
Here is the App.FirePostComponent we use to display an individual blog post:
App.FirePostComponent
<script type="text/x-handlebars" data-template-name="components/fire-post"> <div class="post-header"> <img {{ "{"}}{{ "{"}}bind-attr src="post.user.avatar"}} class="avatar post-avatar"/> <div class="post-author">{{ "{"}}{{ "{"}}post.user.id}} <div class="post-date">{{ "{"}}{{ "{"}}post.publishedDate}} <h3 class="post-title">{{ "{"}}{{ "{"}}post.title}}</h3> <div class="post-content">{{ "{"}}{{ "{"}}breaklines post.body}} <ul class="post-comments"> {{ "{"}}{{ "{"}}#each comment in post.comments}} <li class="post-comment"> <img {{ "{"}}{{ "{"}}bind-attr src="comment.user.avatar"}} class="avatar post-comment-avatar" /> <div class="post-comment-body"> <div class="post-comment-meta"> <strong class="post-comment-author">{{ "{"}}{{ "{"}}comment.user.username}}</strong> <em class="post-comment-date">{{ "{"}}{{ "{"}}comment.publishedDate}}</em> {{ "{"}}{{ "{"}}breaklines comment.body}} </li> {{ "{"}}{{ "{"}}/each}} </ul> </script>
Now we have a full-featured blogging application that displays posts and comments in realtime without a server. It’s that simple!
Our goal with the new EmberFire library is to make the process of adding a backend to your Ember app and integrating with Ember Data as seamless as possible. We encourage you to take our new bindings for a spin. Submit a pull request or email me at adam@firebase.com with any feedback. We’re just getting started with this new Ember Data integration, and we’re working hard to make updates based on your suggestions. We’re excited to see what you build with Firebase and Ember.
For those of you going to EmberConf, look for James, Alex, Sara and I in the bright yellow t-shirts!
Tom Larkworthy is the developer of Firesafe, a language wrapper for Security and Firebase Rules. Firesafe eases development of complex protocols.He cares about logical perfection, and finds holes in Firebases using formal methods.
Data integrity is hard to scale in databases. Issues surface quickly in massively multiplayer games, where the players are incentivized to cheat. Firesafe is a language for expressing and checking protocols to solve data integrity problems. In this post, I show how sending an item between players can be implemented safely and deadlock free with a protocol developed using Firesafe.
Data integrity is very hard to manage in scalable, realtime applications. Most massively multiplayer games have been hit with an item duplication bug at some point. This type of bug allows users to clone rare items, consequently trashing the virtual economy with deflation, ruining the reward structure and costing the hosting company millions in lost revenue. If not managed, the virtual world can be damaged for years. This has affected popular games in the past, including World of Warcraft, Runescape, Farmville, and many others.
The dual of the duplication bug is the bug where items disappear. The loss of an item flushes a player's efforts down the drain, makes them very angry, and usually causes them to call support. The major cost in massively multiplayer games is not the operational cost of running servers, but the cost of answering thousands of support requests. Data integrity problems are a major cause of support requests.
These problems affect big companies with big budgets, and these bugs are costing them millions. So why don't companies just develop working code?
The reason why these bugs are widespread is because of a common underlying technical issues. Essentially, it is difficult to ensure that transactions always work correctly in a distributed system (called ACID compliance). ACID compliant transactions are even harder to implement when servicing a real time system.
Firesafe is a language for expressing state machine dynamics that is compiled into Security and Firebase Rules. State machines are a natural programming abstraction used extensively in the telecommunications industry to solve similar hard problems. State machines are attractive for formal verification because their finite sized state space makes enumerating their total behavior tractable.
I have developed a system that verifies the correctness of protocols implemented with Firesafe. This post is a demonstration of one possible use of this technology: the development of a protocol for ACID-like transfer of items between users. Item transfers are a canonical hard dynamic to get right in multiplayer games.
A lot of bugs in multiplayer games are centered around the trading dynamic, when one player sends an item to another player. What should happen is one player loses an item, and another player gains the item. Often this is not the case.
Common errors are both players end up with the item, or both lose the item. It sounds like an easy feature to develop, but the player accounts might be on different computers, and a player could (deliberately) disconnect from the game at any point.
Distributed exchange of state is a similar problem banks have when performing an electronic transfer between financial institutions. In developing a solution for Firebase, I have drawn from the bank's solution of a 2 phase commit protocol (2PC).
In a 2PC, a central authority coordinates the a transaction between party A and B. In this approach, it is critical that the coordinator waits until both A and B have indicated they are willing to commit before sending a message that the transfer is going ahead (or not). Before the commit message, either party can back out the transaction or go offline, in which case the coordinator must announce that the transfer is aborted through a message.
A and B have to wait for the coordinators decision, which is the main issue with 2PC.It is safe, but it blocks if the coordinator fails. There are also problems if a coordinators commit message goes missing. As we will see, due to some of the guarantees provided by Firebase, many of the main issues of 2PC disappear.
Firebase is like no other database. It is completely scalable and realtime, but has no stored procedures and all data must be in a tree structure. It only supports transactions on sub-trees, and it enforces data integrity through a custom security language, which is less expressive than traditional SQL conditions.
It is also a work of genius.
While Firebase may seem like a long list of restrictions, it is clear that every restriction is very well thought out and provides an opportunity for blazing realtime performance and scalability. So Firebase is a hot rod, and you don't even have to host anything! The Firebase team deals with expensive operational issues, which democratizes access to top DB performance.
That said, the drawback is that ensuring data goes in and out the database properly is more difficult, as there is no opportunity to run secured arbitrary application level code before remote clients access the database layer. You can only manage data integrity with the security functions Firebase provides. In the case of transferring data from one user account to another, one problem is that Firebase does not support transactions across sibling branches.
An often touted route around Firebase's limitations is to add an additional privileged server to provide additional functionality. However, that is not ideal, as now you have to manage those servers, and they also add additional latency.
The true way for item transfer, in keeping with the Firebase philosophy, is to have clients carry out the transactions. By leveraging atomic in-branch updates, plus out-of-branch referencing in Security and Firebase Rules, it is possible to sequence concurrency safe protocols. However, this requires carefully thought out Security and Firebase Rules to ensure the system isn't open to exploitation.
Firesafe is a language that lets us encode state machines with managed variables inside a Firebase. It provides hooks to limit how variables are modified based on the machines current state. This is a natural way of expressing concurrent protocols, and allows developers to concentrate on the "clever bits" of the the protocol without the error prone drudgery of managing state transitions.
Inspired by the banking sector, I’ve designed a protocol that lets one player send an item to another player. This post gives a basic overview of a two-phase commit example; for a full explanation with code samples, check out the github wiki.
Note that each player account resides on a different branch of user. So we can't use Firebase's subtree transactions to make it atomic.
This kind of transaction is business critical. Either it works, or the transaction is rolled back to the original state. No dupes. No losses. Furthermore, we want this system to be deadlock free. If one player goes offline, it should not hold the other player in limbo.
I have taken the classic 2PC protocol and adapted it for Firebase. Firebase is only able to verify data with boolean clauses, it can't actually move data around itself, and it can't emit messages like the classic central authority.Thus, the connected clients have to initiate the changes of state, with Firebase simply double checking the validity.To get safe transfers, we have to logically force the clients down a predefined set of state transitions that can only lead to transfer of the item or an aborted and rolled back transfer.
An sketch of a successful integration of the two players is shown below.
The ordered stages of the protocol is:
We need a number of variables for each user:
We then need to add Security and Firebase Rules to ensure a client can only alter those variables in a way that leads to our item transfer dynamics. To avoid deadlocks, a player should be able to push an authorised transfer through, even if the other party goes offline. The permissions have to be carefully set to avoid one player authorising the transfer on the other player's behalf. Either player should be able to abort before confirmation.
The whole protocol, including deadlock free rollbacks even if one player goes offline, is shown below:
The Firesafe source HSM representation is here, the compiled rules file is here.
Firesafe's HSM representation is 50% less lines of code, and a large portion of that is variable and state declarations. The Firesafe language lets you concentrate on the guards and effects whilst alleviating the burden of enforcing state switching. There are features of the HSM language we have not mentioned in this post. HSM supports Hierarchical State Machines which is a feature that reduces space explosion by inheritance.
When I first considered developing sibling transactions on Firebase, I did not know deadlock-free ACID compliant transfers would be possible. Developing this protocol was my own personal test of Firebase's metal. With Firebase passing this test, I am fairly sure you can do anything on Firebase without involving third party servers.
Getting data integrity right is tough. I was only sure I had it right after putting it through formal verification (which did indeed find lots of holes in my original logic). Applying formal methods to Firebase semantics is not straight forward yet. You need to ask yourself this question, if a malicious user exploits your database, can it financially ruin you, or nuke your product? If the answer is yes, then test driven development is not a robust enough methodology, you need formal verification.
If you are interested in having Tom verify your app’s security design, you can email him at tom.larkworthy@gmail.com to inquire about this kind of service. Stay up to date on his verification technology here.
James, Andrew, Sara and I are back from ng-conf -- and we all had a blast! The conference was very well put together (kudos to Domo) and it was inspiring to speak with luminaries from across the Angular community in person. Just in time for ng-conf, we released a new version of AngularFire (0.6), deployed improvements to our account dashboard, and announced a new Firebase + Angular quickstart guide. I was also excited to demo on stage how easily you can wire up a backend to your Angular app using Firebase.
Watch this video of my talk on building realtime apps with Firebase and Angular:
The conference started off with an amazing keynote by the creator of Angular himself, Miško Hevery, along with Brad Green from Google. They gave a bit of history behind Angular, how it came to be, where they are today and what their plans are for the future, including how they want to help the community extend the Angular feature set. We were extremely flattered when Miško said in the keynote that Firebase “completes the original vision” of Angular:
After a long day of excellent talks by several preeminent speakers, it was time for the Firebase sponsored party. The party featured a comedian, magician and an impromptu comedy troupe. A spread of desserts was accompanied with our very own Firebase chocolate. We hope everyone had a great time - conferences are so much better when they’re mixed in with a little live entertainment!
We gave away tons of Firebase swag: T-shirts, light-up bracelets, stickers, and of course, our very own hot sauce! We were very happy with all the interest in Firebase, and were delighted to meet everyone who came to our booth with questions, compliments or feedback.
We wrapped up the conference with a day on the slopes with many other ng-conf attendees. It’s one of the best conferences we’ve participated in so far, and we’re definitely looking forward to the next one!
Ever since we launched our Firebase bindings for AngularJS last March, we’ve been amazed by the Firebase and Angular apps we’ve seen from our community. This year, we’re incredibly excited to be a Platinum sponsor at the first ever Angular conference, starting this Thursday in Salt Lake City, Utah! We’re also happy to announce our new Angular + Firebase Quickstart Guide, which makes it even easier to get started with Angular and Firebase.
If you won’t be at the conference, head over the ng-conf home page for a link to a livestream of all the talks. You can also follow @Firebase and #ngconf on Twitter for the latest updates. The ng-conf organizers have pulled together a great lineup of events. Here’s where you can find us at the conference:
Hear the latest on Firebase + Angular, including a special announcement: On Friday at 10:35am MST (GMT -7), I will be speaking about the powerful combination of Firebase and Angular, and how the two tools are changing the future of modern web apps. I’ll also be making an announcement about Firebase.
Ask questions at Firebase office hours: For those of you attending ng-conf, get your technical Firebase + Angular questions answered in person and share your apps with the Firebase team! We’re holding office hours on Friday 1/17 from 7:30 - 9am. Sign up in advance or drop in at the conference.
Firebase Sponsoring ng-party: We are the sponsor of the official ng-conf party on Thursday night (1/16). The party will feature performances from a comedian, an illusionist, an improv group, and there will be some delicious desserts.
Firebase + Angular Quickstart Guide: Our new quickstart guide provides all the resources you need to start building realtime apps with Firebase and Angular.
Firebase + Angular Starter Pack: To see all of our example apps built with Firebase and Angular, simply clone our Angular Starter Pack on GitHub. The examples are a great place to start playing with code and learning about the integration.
The Complete Angular Guide: If you’re new to Angular, we recommend ng-book, a comprehensive guide to AngularJS with a chapter on adding a Firebase backend to your app.
Firebase + Angular Google Group: Join our Firebase + Angular Google Group to ask technical questions or share your apps with the community.
James, Andrew, Sara and I are looking forward to seeing many of you at ng-conf!
Update (November 4, 2014): While this post still contains some useful and relevant information,we have released advanced query functionality which solves a lot of the problems this post discusses.You can read more about it in our queries blog post.
In Part 1 of this post, we covered a series of common SQL queries and how they can be recreated in Firebase, building off our authoritative Denormalizing is Normal post from last year. We're going to build on several principles introduced in those articles, so be sure to check those out before digging into this post.
In this tutorial, we'll explore a fast and powerful approach to performing text searches, or content searches, in Firebase.
Why Not Just WHERE foo LIKE '%bar%'?
The 20 year old SQL paradigm for content queries (WHERE foo LIKE '%bar%') is a staple for static databases, but not a simple prospect for real-time data. Our team is hard at work on a series of tools to bring content searches into the lightning-fast realm of Firebase's NoSQL data store. Look for more news on these indexing and query-related tools in the coming months.
Until then, I'd like to introduce you to a few quick scripts that can add powerful content searches to your app. At the end of the article, I'll share a library that incorporates these strategies into a service you can clone, configure, and run on your own box.
Introducing ElasticSearch
ElasticSearch, based on Lucene,is an extremely powerful document storage and indexing tool. However, at its core is a very simple search feature, which is nearly plug-and-play compatible with Firebase.
While it's certainly not the only way to write content searches in Firebase, ElasticSearch's simple integration makes it fast to implement and takes very little code to utilize, while its powerful indexing capabilities provide for customization, allowing it to scale with your app.
You can set up a local instance for testing in three steps:
It's really that simple! And, surprisingly, deploying a free, hosted instance requires nothing more than a button click thanks to Heroku's Bonsai add-on.
Indexing Firebase Data
The first step is to get data into ElasticSearch so it can be indexed. A simple Node.js script can plug Firebase into ElasticSearch with a few lines of work. I utilized the node-elasticsearch-client library, which is optional, but simplifies the process by wrapping the lower level ElasticSearch client:
var Firebase = require('firebase'); var ElasticClient = require('elasticsearchclient') // initialize our ElasticSearch API var client = new ElasticClient({ host: 'localhost', port: 9200 }); // listen for changes to Firebase data var fb = new Firebase('<INSTANCE>.firebaseio.com/widgets'); fb.on('child_added', createOrUpdateIndex); fb.on('child_changed', createOrUpdateIndex); fb.on('child_removed', removeIndex); function createOrUpdateIndex(snap) { client.index(this.index, this.type, snap.val(), snap.key()) .on('data', function(data) { console.log('indexed ', snap.key()); }) .on('error', function(err) { /* handle errors */ }); } function removeIndex(snap) { client.deleteDocument(this.index, this.type, snap.key(), function(error, data) { if( error ) console.error('failed to delete', snap.key(), error); else console.log('deleted', snap.key()); }); }
Drop that in a hosting environment like Heroku or Nodejitsu, or onto your own host with forever to monitor up-time, and search indexing is done! Now it's time to read some of that data back.
A Brute Force Search
Once we have our data indexed in ElasticSearch, we could directly query our index using a wrapper like elastic.js. This is a perfectly reasonable option, but does add coupling and dependencies to the client:
<script src="elastic.min.js"></script> <script src="elastic-jquery-client.min.js"></script> <script> ejs.client = ejs.jQueryClient('http://localhost:9200'); client.search({ index: 'firebase', type: 'widget', body: ejs.Request().query(ejs.MatchQuery('title', 'foo')) }, function (error, response) { // handle response }); </script>
Since our clients are already using Firebase, wouldn't it be great to keep our client code agnostic and push the request to Firebase instead?
A Firebase Search Queue
This little node script listens at /search/request for incoming searches, handles the interactions with ElasticSearch, and pushes results back into /search/response:
/search/request
/search/response
var Firebase = require('firebase'); var ElasticClient = require('elasticsearchclient') // initialize our ElasticSearch API var client = new ElasticClient({ host: 'localhost', port: 9200 }); // listen for requests at https://<INSTANCE>.firebaseio.com/search/request var queue = new Firebase('https://<INSTANCE>.firebaseio.com/search'); queue.child('request').on('child_added', processRequest); function processRequest(snap) { snap.ref().remove(); // clear the request after we receive it var data = snap.val(); // Query ElasticSearch client.search(dat.index, dat.type, { "query": { 'query_string': { query: dat.query } }) .on('data', function(data) { // Post the results to https://<INSTANCE>.firebaseio.com/search/response queue.child('response/'+snap.key()).set(results); }) .on('error', function(error){ /* process errors */ }); .exec(); }
A Client Example
Now that we have a way to queue requests into Firebase, the client can simply push requests and listen for results:
<script> var queue = new Firebase('https://<INSTANCE>.firebaseio.com/search'); function search(index, type, searchTerm, callback) { // post search requests to https://<INSTANCE>.firebaseio.com/search/request var reqRef = queue.child('request').push({ index: index, type: type, query: searchTerm }); // read the replies from https://<INSTANCE>.firebaseio.com/search/response queue.child('response/'+reqRef.key()).on('value', function fn(snap) { if( snap.val() !== null ) { // wait for data snap.ref().off('value', fn); // stop listening snap.ref().remove(); // clear the queue callback(snap.val()); } }); } // invoke a search for *foo* search('firebase', 'widget', '*foo*', function(data) { console.log('got back '+data.total+' hits'); if( data.hits ) { data.hits.forEach(function(hit) { console.log(hit); }); } }); </script>
A Pre-Built Library for Your Use
We've implemented a content search for Firebase using ElasticSearch, set up a queue to ferry data transparently to and from our search engine, and finished all of this in a couple short scripts. We've also tapped into the powerful, flexible, and scalable world of ElasticSearch, which will grow with our app.
Easy enough? I've gone ahead and baked these scripts into a GitHub repo just to make things even simpler: Fork the Flashlight service on GitHub.
It's MIT Licensed and ready for cloning. Just edit the config file, start it up, and get back to work on your app!
We want your feedback!
Have fun and let us know how we did! We love getting feedback from our dev community. Chat with us in the comment thread or send an email to wulf@firebase.com. You can also follow @Firebase on twitter.
