Trying to choose a database technology for your application? It can be really tough. The database market is evolving rapidly. From NoSQL, to NewSQL, to the mature relational systems, there's no shortage of solutions to sort through.
FoundationDB offers some great ways to interact with the core, key-value (KV) store in your application’s native language. So, for example, if you use Java in your application code, you can use our Java client binding that implements the database functionality in ways that are idiomatic to Java. There are also existing bindings for C, Python, Ruby, Node, Go, PHP, .NET and even layers and bindings from our community ecosystem.
But what if you wanted to use the KV store in a way that’s language independent and flexible for your applications? It turns out that building a basic REST layer on top of the KV APIs is straightforward to implement when you follow some of the basic design patterns that we outline in our Design Recipes. Of course, the difference here is that you’re exposing an entirely different API for your applications, but the same general ideas apply.
TL;DR: SQL Layer is an open-source Google F1 style fault tolerant and scalable SQL database, available as a free Layer for the FoundationDB Key-Value Store.
Good News! FoundationDB is now available in the AWS Marketplace. Now you have another easy way to spin up FoundationDB in addition to the AWS CloudFormation Template available on our site. It’s available as a ready-to-deploy AMI that can be used to form a full cluster for your environment in just a few minutes.
Hi, I'm Rishub, an intern at FoundationDB. I am a rising senior at Forest Park High School in Woodbridge, Virginia. I've been working here since June, and it's been a blast so far. One of my first tasks was to create a web app to demonstrate the capabilities of the database. For this project, I had to learn a new language, Ruby. Now, I am a C/C++, Java type of guy, and web development is a bit new to me. It was difficult at first to get used to Ruby's idioms and dynamic types, but I managed to pick it up fairly quickly by working on translating some of FoundationDB's Design Recipes from Python to Ruby.
A few days ago I sat down and chatted with Archie Cobbs about JSimpleDB, his open-source Java ORM.
Hi, I’m Nikhil Palekar, and I run the Solutions group at FoundationDB. Recently, we’ve had lots of customers reach out to us to discuss modeling time-series data in FoundationDB, so I thought I would write a quick blog post about time-series data and how it fits into FoundationDB.
Great news! Today, we announced a partnership with DataRPM, a cognitive big data discovery and analytics platform. What does it mean for our users? The integration will allow you to perform data analysis and discovery on data stored in a distributed, transactional SQL database (via the FoundationDB SQL Layer). This is an important step in solving two of the biggest challenges in the data management industry today: data is getting bigger and moving faster than ever before and expectations of application uptime have never been higher.
You won’t have to look hard to find examples of projects built with FoundationDB. The ecosystem page has 20 – from quotes to case studies to GitHub repositories. Everyday people ask questions on Twitter, our QA site and Google Groups about the problems we’re helping them solve, and a search for FoundationDB on GitHub returns 59 results.
There’s a lot of code to peruse out there, but I think what’s missing is the individual. Who are the people who make up the FoundationDB community? What motivates them to champion a new database to their CTO? Why are so they so fired up they’re pushing commits to open-source projects at midnight? And what is with the evangelist who, beer in hand, pesters everyone at the NoSQL conference with his love of ACID transactions? These are the interesting stories of FoundationDB.
This is the first post in our new Developer Spotlight series. Each one will focus on developers who have put something out there for us, from source-code to conference talks. Let’s get to know the FoundationDB community.
For this first post, I sat down with James Thompson to chat about Hypostasis, his project for providing an idiomatic Ruby API for familiar and experimental data models on FoundationDB.
Hello, I’m Alec, one of three interns working at FoundationDB. I’m currently studying at the University of Virginia, but I’m home for the summer and excited to be working here. As part of my duties, I was tasked with making some of our pedagogical material available in Java in addition to Python as both an introduction, for me, to the FoundationDB key-value store API as well as to give other developers the chance to see more material in different languages.
We’ve just added a new course on using Concurrency with FoundationDB. Like all of our online courses, it’s free and browser-accessible. The course is targeted to developers who are familiar with the FoundationDB Key-Value Store API and want to achieve high throughput in their applications. If you’re new to FoundationDB or the Key-Value Store API, you can get up to speed by first taking the introductory course on Building FoundationDB Layers.
I read another "explanation of the CAP theorem" today that was totally wrong. Are we still doing this in 2014? (Pro tip: If an article includes a diagram with a triangle, stop reading). There are also a lot of good explanations but they tend to be quite long. Here's my humble attempt at an explanation of the fundamental CAP theorem trade off that is both correct and short:
If you’ve built an application that stores its data in a relational database like MySQL or PostgreSQL, then you’ve likely run into situations where joining two or more large tables becomes very slow and painful. Because our SQL Layer stores its underlying data in our Key-Value Store, it has a unique feature called “Table Groups” that alleviates painful joins and allows your application to stay speedy.
Many applications have objects with a hierarchical relationship. Imagine an e-commerce app using a relational database that has an orders table, a customers table, an addresses table, and an items table.
A few weeks ago I had the pleasure of attending NoSQL Matters in Cologne, Germany. The first night of the conference all of the attendees and speakers came together for chili con carne and Kolsch beers, and I asked many people what their favorite talk of the day was. The most popular response I heard was the keynote, ‘The Next Horizon of NoSQL’ by Ted Dunning. Ted’s talk was a forward look at what developers need, and should demand, from database vendors in the near future. Here’s a great summary slide of his suggestions:
Last week, Jen and I were in Germany and The Netherlands for NoSQL Matters and our first ever FoundationDB Amsterdam meetup. Both of which were great: we met a lot of awesome people, heard interesting discussions on the current landscape and future of NoSQL, and even got to introduce engineers to FoundationDB.
We've recently added a new section on Design Recipes to the FoundationDB website. If you want to get a sense of how flexible the Key-Value Store is, check it out!
I am pleased to announce that the SQL Layer has entered the Beta phase.
You know, the first thing that stood out for me this time here at the conference is that MySQL has now officially entered the "mature product" phase. For example, one of the first slides Tomas, MySQL VP of engineering, showed in his presentation included a rolling list of MySQL products that hardly fit the screen (with a size 10 font). There's a lot of focus on tools and small optimizations. MySQL is getting polished, but the excitement that I remember from the Drizzle or Monty Program, or even the Sun/Oracle acquisitions is gone. It's mature.
Developer Aphyr made waves in the database community last summer when he released a series of blog posts titled "Call Me Maybe" that revealed fault tolerance problems in several databases, including MongoDB and Riak. Aphyr’s open-source durability testing tool, Jepsen, found that in the face of a network partition, some databases will acknowledge writes but then lose data. That’s pretty upsetting when you expect your database to keep the data it says it stored.
We obviously like to brag about our products here at FoundationDB, but we put our money where our mouth is–so to speak–and “dogfood” them whenever we can. So it’s no surprise that our website runs on top of our SQL Layer and core storage substrate. While the site doesn’t currently generate a crazy database load, I thought it was worth showing how FoundationDB gives you the confidence to scale your SQL application on top of a fault-tolerant database.
We deploy everything on Amazon EC2 as two logical layers: the app tier and the storage tier.
Hi there! Stephen here. I just got back from speaking at the DC Python Meetup, and I thought I'd share some of the great discussion from the session.
Last week Happy Pancake’s Rinat Abdullin dropped a little teaser to his twitter followers:
Just had an awesome conversation with @jonathan_oliver about software design, @happypancakerd and @FoundationDB. It was recorded ;)
— Rinat Abdullin (@abdullin) February 20, 2014
Rinat has been generous in sharing his experiences with FoundationDB on twitter, so I was really excited to learn about his work in more than 140 chars. Fortunately he didn’t keep me waiting for long, as the podcast was announced a few days ago. Here’s the link to the full episode.
Hey, it’s Jen, and I’m back to continue the discussion on FoundationDB’s correctness testing strategy. My first blog in this series compared FoundationDB to Netflix’s Chaos Monkey, including similarities and differences, and explained the variety of failure scenarios each test.
To reiterate, fault tolerance is important to us because FoundationDB is designed to be ACID compliant in a clustered environment, where the chances of something going wrong are higher than with a single machine.
We’re very excited to introduce FoundationDB 2.0. FoundationDB combines the power of ACID transactions with the scalability, fault tolerance, and operational elegance of distributed NoSQL databases. This release was driven by specific customer feedback for increased language support, network security, and higher-level tools for managing data within FoundationDB.
FoundationDB 2.0 adds Go and PHP to the list of languages with native FoundationDB support. There also are two new layers available in all languages: The Subspace layer provides an easy way to define and manage subspaces of keys via key prefixes. The Directory layer manages the efficient allocation and management of virtual “directories” of keys and values within a database. They work together as the recommended way to efficiently organize different kinds of data within a single FoundationDB database.
Hi there, it’s Jen. Now that I’ve settled in at FoundationDB (you can read about my first week here), I thought it would be interesting to highlight what goes on behind the scenes at FoundationDB in terms of our strict testing and simulation regime in a series of blog posts.
If you’ve been following FoundationDB for a while, you know that we’re passionate about designing a database that can maintain ACID guarantees in the face of catastrophic failures. We tote our demo cluster across the country, asking strangers to try and break our database by turning off machines and unplugging network cables. (Missed us in person? You can check out a video here).
Hi there, I'm Jen, the newest member of the FoundationDB team. You know, starting a new job is a wonderful experience. You get to meet new people and push yourself to learn new technologies. But it's also intimidating - all those new people to meet, all those new things to learn!
It’s been five months and a few cycles of integrating the SQL Layer, formerly known as Akiban Server, to FoundationDB. We now have a fairly robust, distributed SQL Layer. For those of you new to the SQL Layer, it is a scalable OLTP SQL system built on the FoundationDB storage substrate. It inherits the substrate’s performance, multi-node scalability, high availability, fault tolerance, and multi-key ACID transactions, and adds a sophisticated SQL environment with direct object access.
Let me take you on a quick tour of what's been happening under the covers:
Let’s take a look at a Celery layer written by our summer intern, Abi Gopal. Celery is an open-source, distributed task queue implemented in Python that allows for easy task creation and scheduling. It can be configured to use a database for its message broker and/or as a backend to store results.
The basic execution units in Celery are tasks, which are distributed for concurrent execution by worker processes. Celery exposes a task decorator that transforms a regular Python function into a task.
Recently, someone brought to my attention a blog post that benchmarks FoundationDB and another responding to the benchmark itself. I'll weigh in: I think this benchmark is unfair because it gives people too good an impression of FoundationDB's performance. In the benchmark, 100,000 items are loaded into each database/storage engine in both sequential and random patterns. In the case of FoundationDB and other sophisticated systems like SQL Server, you can see that the performance of random and sequential writes are virtually the same; this points to the problem. In the case of FoundationDB, an "absorption" mechanism is able to cope with bursts of writes (on the order of a minute or two, usually) without actually updating the real data structures holding the data (i.e. only persisting a log to disk, and making changes available to read from RAM). Hence, the published test results are giving FoundationDB an unfair advantage. I think that you will find that if you sustain this workload for a longer time, like in real-world usages, FoundationDB might be significantly slower.
Welcome back to our ACID whiteboard series! In these videos we’ve discussed database basics and talked a bit about FoundationDB. If you haven’t already, check out the first three videos in this series on the “A” in ACID, Atomicity, the “C” Consistency, and “I” Isolation. This is the last video in this series on “D” for Durability.
I am pleased to announce today that FoundationDB has raised $17 million led by Sutter Hill Ventures and with additional funds from our existing angel investors and CrunchFund. This exceptional level of investment reflects the exciting things that you, our customers and users, are doing with FoundationDB, as well as confidence in what our talented team can continue to build.
Hi there! I'm David Browning. We probably haven't met before, and that's too bad. I design, develop, and manage our web sites/apps/properties/tubes here at FoundationDB. I'd like to kick off the first of hopefully many posts to come about our company culture here at FoundationDB.
This weekend GigaOM published a piece called "Next gen NoSQL: The demise of eventual consistency?" by my co-founder Dave Rosenthal. Generally, the response was positive and many people (including Alex Popescu) said it was well written. However, on his blog Alex had some criticisms of the article. Here, I'll respond to them.
Welcome to our whiteboard series. In these videos we'll dive into database basics and talk a bit about FoundationDB.
This is Part 3 of a 4-Part introduction explaining a bit about the "ACID" properties of database transactions. If you haven’t already, check out the first two videos in this series on the “A” in ACID Atomicity and “C” Consistency.
The rise of distributed databases, spanning both NoSQL and older relational technologies, has produced all the confusion typical of fast-moving fields whose engineering designs have not yet matured. Developers now face many choices when deciding how to store and manage their data, although the available solutions differ greatly in their performance, scalability, and flexibility.
This screencast walks through configuring FoundationDB server processes on a single machine. We explain how to configure the number of processes and tune settings to take best advantage of your hardware. We use a Linux system but you can check out our docs on configuration for Mac and Windows, which is nearly the same.
Welcome to our whiteboard series. In these videos we'll dive into database basics and talk a bit about FoundationDB.
This is Part 2 of a 4-Part introduction explaining a bit about the "ACID" properties of database transactions. If you haven’t already, check out the first video in this series on the “A” in ACID, Atomicity.
Welcome to our whiteboard series. In these videos we'll dive into database basics and talk a bit about FoundationDB.
We put together this screencast to walk through installing and scaling your FoundationDB cluster beyond the default single-machine installation. In it, we explain how to add two more machines and configure the cluster to take advantage of the high performance and fault tolerance properties of FoundationDB.
We’ve been talking a lot about transactions lately, and I wanted to amplify a few of the key points of the conversation.
The real usefulness of transactions comes from their ability to coordinate and synchronize a variety of data elements when multiple clients perform updates concurrently. For transactions to play this role, the developer must have the ability to specify whatever data elements are logically needed. A system may implement more limited operations, like Compare And Swap (CAS), with one or more of the ACID properties, but these operations aren’t transactions.
Cassandra 2.0 was released a few days ago, and its headline feature is a compare-and-swap operation. I’m happy to see Cassandra added CAS, which is a very useful capability -- that makes the world a little better. And the database world needs all the “better” it can get. I’m less happy to see the feature referred to as “lightweight transactions”, because that is misleading and makes the world a little more confusing. And the database world does not need any more confusion.
Compare-and-swap is not a general transaction facility. At bottom, transactions provide the ability for an application to do multiple operations “together” without interference from other concurrent access to the database. Compare and swap provides a special case of this: the ability to do two operations “together”, one of which is a read from a row (or “partition” in Cassandra), and the other a write to the same row. Being a special case of a transaction is not very exciting in and of itself, because transactions are so general: just about anything is a special case of a transaction.
At the NoSQL Now conference in San Jose, I presented the case for why I believe future generations of NoSQL databases will need to support ACID transactions in order for developers to more easily build, deploy and scale applications.
The Google F1 database paper just became generally available. It’s the database that runs AdWords, so I suspect Google kind of cares about it. Google F1 bears an amazing resemblance to our FoundationDB SQL layer. Both leverage an underlying scalable and transactional storage substrate to allow unprecedented scalability at both the storage and SQL processing levels. Both leverage hierarchical schemas, arranging tables into a hierarchy that is mirrored in storage by interleaving rows of parents with rows of children. Both systems make the tradeoff that higher latency is acceptable for gaining almost unlimited bandwidth and scale. A query may require more round-trips and take a (bounded) longer time, but the overall throughput of the system, measured in number of queries that can run in parallel, is much higher.
The fact that Google has built and is using F1 for their AdWords business demonstrates how the architectural approach works for even the most demanding of applications. However, F1 is only available to Google’s internal teams, while our open-source FoundationDB SQL Layer is intended to have a somewhat broader audience.
After a successful 18-month Alpha and Beta testing program involving more than 2,000 participants, we're very excited to announce that we've released version 1.0 of FoundationDB and general availability pricing!
As we prepare for our GA launch, we are happy to announce the final beta version of FoundationDB, Beta 3. You can download Beta 3 here.
A few weeks ago we posted our Fault Tolerance Demo Video and it got a lot of interest, making it to the top of Hacker News for most of a day. We figured since our demo cluster video was interesting to people, we'd show off an actual fault tolerance testing tool we use here at FoundationDB - the worst, least reliable database cluster we could put together, aptly named Quicksand.
Let’s take a look at the string interning layer, one of our example layers available on GitHub. Each of these simple layers supports new functionality with a small amount of code that is easy to inspect, extend, or modify.
Layers add a wide variety of capabilities to FoundationDB, such as new data models, high contention data structures, and compatibility with legacy systems. The string interning layer is an example of a utility, a lightweight layer that solves a specific task with a simple API.
We’re happy to announce that we've released FoundationDB Beta 2!
Most of our testing and tuning in the past has focused on data sets ranging up to 1TB, but our users have told us that they’re excited to begin applying FoundationDB’s transactional processing to data sets larger than 1 TB, so we made that our major focus for Beta 2.
Let’s continue our look at the example layers we’ve made available on GitHub. These simple layers add significant capability to FoundationDB while inheriting its transactional, scalable, fault-tolerant properties.
In this post, we’ll look at the counter layer, which allows multiple clients to share a high-contention counter while minimizing transaction conflicts.
Below is a video showing off the fault tolerance and recovery properties of FoundationDB under some nasty conditions (single & multiple machine failures, network partition, full cluster power loss) on a live demo cluster we bring with us to events:
Over the next few weeks, we’ll take a look at some example layers (all available on GitHub). These simple layers add significant capability to FoundationDB while inheriting its transactional, scalable, fault-tolerant properties.
Let’s start with the spatial indexing layer for working with two-dimensional (2D) point data. FoundationDB’s layer concept lets us implement spatial indexing outside the database with a small amount of code, making it easy to inspect, extend, or modify.
Back in January, I was asked a question on Quora: "Why did FoundationDB take so long to come out of stealth mode?". It seemed like a good question to answer for people due to the surprised reaction we get from many in the startup ecosystem when we tell them we were in stealth development for 2.5 years, and had a selective closed alpha testing phase for an additional year. This was a conscious decision on our part, and we believe it is one of the reasons our technology is so well positioned today.