Google BigQuery Adds Streaming

Google’s cloud service, BigQuery, received a big update this week as the search giant announced that they’ve added a new streaming feature to the data crunching tool (along with a sprinkling of other upgrades).

“BigQuery is now able to load and analyze data in real time through a simple API call, the new tabledata().insertAll() endpoint,” wrote Google Developer Programs Engineer, Felipe Hoffa announcing the changes. “This enables you to store data as it comes in, rather than building and maintaining systems just to cache and upload in batches. The best part? The new data is available for querying instantaneously. This feature is great for time sensitive use cases like log analysis and alerts generation.”

To get developers familiar with the new capability, Google says that they will be offering the service for free for the rest of the year. Once the introductory period ends on January 1^st, 2014, the service will be billed at a flat rate of 1 cent per 10,000 rows inserted. Google says that the traditional jobs().insert() method will continue to be free.

In addition to the ability to load and analyze large amounts of data as they are delivered, Google has also added the ability to query specific ranges or subsets in the data, like the last 24 hours, rather than needing to do a costly full column scan – saving their customers money. “Querying a partial subset of data with these decorators will result in lower querying costs — proportional to the size of the subset of the queried data,” wrote Hoffa.

Other features added to the release were new window and statistical functions, adding SUM(), COUNT(), AVG(), MIN(), MAX(),FIRST_VALUE, and LAST_VALUE(). BigQuery’s window functions enable users to perform calculations on specific partitions, or “windows” of a results set, giving them powerful tools for ranking and exploring their data without having to self join.

Last, but not least, Google has added a new BigQuery browser tool, that analyst can use to query their history faster, using a Query History panel. The tool comes with a slick interface aimed at making information about past queries more convenient and easier to extrapolate on.

Related items:

LinkedIn Open Sources Samza Stream Processor

Apache Takes Storm Into Incubation 

Twitter Conjures Up a Hadoop-Storm Hybrid, Ponders IPO