[Portland, the United States, July 20, 2015] Huawei announced today at O'Reilly Open Source Convention (OSCON) that Spark SQL on HBase package is an open source and available. Dubbed Astro, the end-to-end package combines the capability of Spark, Spark SQL and HBase, helps drive Spark adoption in broad NoSQL customer base and provides powerful online query and analytics capability for large scale data processing in vertical enterprises. Huawei has embedded Spark into its big data solution, FusionInsight, which is used by the world’s largest bank, Industrial and Commercial Bank of China, and communications service provider, China Mobile. Huawei will also launch Spark as a cloud service in China on Huawei Public Cloud this year.
Astro has new features that enable systematic and powerful handling of data pruning, intelligent scan, and pushdowns like custom filters and coprocessor, and make more traditional RDBS capabilities possible.
The algorithms for query optimization are generically applicable to organized data sets and can be utilized by distributed SQL engine. To leverage latest Spark features including Dataframe and external data source APIs, Huawei global team has kept the package up with the changes and makes it in compliance with new Spark 1.4 release.
It has attracted community contributors being involved in development and validation works. Besides the Astro project, Huawei team has contributed new features to Spark SQL, Machine Learning and Spark R to help enrich standard libraries in Spark.
“Open source is Huawei’s company-wide strategy, from core business networking, where the firm helped drive the network openness as founding member of OPNFV, to new business frontiers cloud computing and IoT, where we open sourced LiteOS, the world’s most lightweight IoT Operating System to standardize and simplify the infrastructure and enhance the IoT connectivity” said Wang Chenglu, President of Huawei Central Software.
“Spark, as the most influential community among Apache projects and as preeminent framework of data processing and analytics, has been our focus and top priority. We believe that putting Spark as the core to our big data solution can bring significant benefits to our customers, keeping up with technology innovations driven by the ecosystem while maximizing their IT investments from the adoption of best practices and other customers’ success,” added Wang.
In China Unicom, one of the top 5 mobile carriers in the world, Huawei deployed Spark in its operating branch across mission-critical business areas. The system supports near real-time query and analysis over multiple data sources, and allows analysts and data scientists to build models over large data sets more effectively and in simpler way, in some cases from months to mere weeks. The deployment has also had success in leveraging Spark to plan recommendation and churn prediction. Huawei and Unicom are working together to spread Spark into new business areas.
The other recent success at one of largest telcos in South America proves that Spark and Huawei big data solution can bring much more business value than traditional BI systems and previous data management platform. The customer gained centralized and up-to-date customer insights 10 times faster, achieved much higher campaign conversation rate in selected services, and developed new business cases in agile and fast ways. Data analysts no longer need to do complex coding on MapReduce and the ML model size and expressivity improved exceedingly.
“We're thrilled to have Huawei as a major contributor to Spark. As a leader in emerging markets and in the telecom industry, Huawei has played a critical role in expanding Spark's footprint and influence.” said Patrick Wendell, Co-Founder of Databricks and Spark Release Manager.
“Huawei has supported Spark through a variety of contributions since the very early days of the project. Looking forward, we expect Huawei to continue both developing Spark and driving its adoption in global vertical markets.”
Spark rose to prominence within the big data world, as it provides a robust programming framework, rich set of APIs and libraries, vibrant ecosystem, and unparalleled pace of technology innovation.
“Huawei’s global team is in the process of bringing business case driven new algorithms into MLlib, while exploring the possibility to work with Databricks and the community on vectorized processing and SQL optimizer as part of core project Tungsten,” said Bing Xiao, Head of Big Data at Huawei America R&D Center.
“We have seen community interests on Astro and will collaborate with other vendors to enhance it and spread its adoption. In addition, there’s emerging customer need of Spark SQL on OLAP Cube and we’re evaluating the community acceptance level to its open source. In short, Huawei is deeply committed to Spark and intends to participate extensively in joint community and industry efforts,” said Bing.